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(57) A method for the recording and retrieval of 
information relevant to the activities of the user. 
The invention may be employed to provide a 
memory prosthesis in the form of a highly port- 
able device (30) (e.g. a device worn like a watch) 
which records all kinds of information about the 
user's life, and provides a simple and rapid 
means of access to that information. The device 
receives and stores information from external 
sources ('objects') via, for example, a cellular 
communications network, or from signal 
generators identifying specific machines, 
places, or people carrying other ones of the 
same type of device. Events comprising en- 
counters between 'objects' are recorded auto- 
matically, being time-stamped and stored 
chronologically, and a rapid search method for 
such a database is disclosed. The data can 
either be stored in the memory of the device 
itsetf, or, for better protection against failure, in 
a central data repository of the system, with a 
fail-safe backup. 
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This invention relates to a syst m and method for th recording and retrieval of information relevant to the 
activitiesoftheuser.Theinventi nisparticularlysuitableforimplementationintheformofamemorya.ddev.ee 

which is portable, and preferably wearable by the user. _ _ 

. Memory problems are widespread but little has been done to exploit .nformat.on technology to help. People 
have many procedures designed to compensate fortheirfallible memory, for example, writing lists, organising 
things in files, putting notices on notice boards and setting alarms. Even in a well-organised settmg. things 
might be mislaid or forgotten because they were not seen to be important at the time they were encountered 
These failures of memory have a cost, both in the time spent recovering (such as lookmg for m.sla.d docu- 
ments) and in the time spent organising things in the first place so that problems do not anse 

One current form of memory, aid is the electronic personal organiser. wh.ch .s s.m.lar in form to a hand- 
held calculator. Although such a device is capable of holding large amounts of information each .tern must be 
entered, usually manually, so the user must both input the information and remember to do so 

Various other systems have recently been proposed to help the user recall past events as well as to prov.de 
reminders of future intentions. For example, in EP-A4>;495,622 there is described an acfvity-based I mforma- 
tion retrieval system which automatically captures and records certain aspects of the context in wh.ch a re- 
corded event took place. The system relies on a variety of apparatuses for monitoring, recordmg and t.rne- 
stamping various key aspects of human activity, from which a readable list of episodes, or acUvrty d.ary can 
be generated automatically. In one example, the system uses encoded ident.f .ers. each .ntended to be earned 
by people working in the environment being monitored. One particularly convenient form of identifier b the 
-aciive badge" offered by Olivetti. This identifier takes the form of miniaturised circurtry .n a housing able to 
be pinned or clipped to the clothing of the person to whom the identifier is Issued. The circuitry is des.gned to 
emu pulse width modulated infra-red coded signalsfor a tenth of a second every 12 to 1 5 seconds. The s.gnals. 
by their nature; have a range of about six metres, and will not travel through walls, wh.ch makes them very 
useful for keeping track of people moving between rooms in a normal working environment ^ In each of the 
rooms or corridors of interest would be positioned one or more sensors responsive to the infra-red s.gnals. 
The sensors would be connected to a masterstation processor, which would have the task of poH.ng the sensors 
for identification sightings"; processing the data (which would include decoding the signals to determine wh,ch 
person was within range of a specified sensor at the time of polling), and present-ng the P™*^^ 
visual format in the form of a diary. This diary, which correlates quite closely with human recollect,ons of past 
episodes, can be used simply to stimulate further recall, or to identify an episode and .Is time of occurrence, 
or to show a set of episodes from different sets. The time may be used to locate the corresponding. terns ,n 
any other set of time-stamped data, such as a sequence of video frames or an audio record of a rneetmg 

A related system was disclosed in an article in The Sunday Times, 11 February 1990, enttfed "Computers 
acquire social skills", by Jane Bird. This mentioned the use of identity buttons to enable the computer to know 
who was where in the building at all times, Associated with the system is a network of microphones and/or 
video cameras capable of recording conversations. It gave the example of a copier .n a room mom tared by the 
computer: Upon the copier's developing a fault, a frustrated would-be user tries to recall whata colleague told 
him about how to cure the fault The computer is interrogated tof ind the last time that the two people concerned 
were at the copier. Once this particular bit of information has been recovered, it « used to access the t.me- 
stamped audio records of what was said in the room containing the copier at that point of time: In tins way, the 
enquirer has the opportunity of hearing again the conversation between him and hs colleague 

The aforementioned systems have several practical limitations. These include the need on the part of the 
usertodevotea lot of effort to identifying a particular time and placefromtherawdata.andthelackofprec.se 
episode definition from such data. Also the systems capture just the location of users; and not their actrvrty or 
other significant episodes. These drawbacks would make the systems of relatively little pract.cal value 

There is still, therefore, a need for an improved memory aid system which requires minimum input rom 
a user and yet which is easierto use for retrieving information than to remember orf ind whatever rt .s that has 
been forgotten or mislaid. It is an object of the present invention to meet the need. 

According to the present invention, there is provided an information recording system compns.ng: 

a plurality of objects, each object representing a human user or other entity .dentifiable by the system 
and each object comprising an agent, means for communicating between the^agent and the agents of other 
ones of the objects, and, in communication with the agent, at least one of a database, a sensor, and a user 

th' syst m including at least on database, at I ast one sensor, and at least neus r interface; 

at least part of th ag nt of at least one of th bjects representing a human user being portabl byth 
user, providing int raction with th user, and identifying to the system the I cation of the us r. 

th information comprising, for a given object, a chronological sequence of ^stamped "£ 
database, each fact comprising an attribut and a value of that attribute for a grven object at the tame .nd.cated 
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by the time stamp; and 

the facts being automatically recorded in th database n the occurrence of physical or logical coll ca-_ 
tions of two or more of the objects. 

According to another aspect of the invention, there is provided a method of searching a database in which 

5 data is organised as a set of threads, each thread comprising a chronological sequence of time-stamped facts 
all of which relate to a single object and at least some of which also represent descriptors of other objects, 
each object representing a person or an identifiable entity, and each of the facts comprising an attribute and 
a value of that attribute for a given object, the method including displaying the sequence of facts of a selected • 
thread, or the sequence of events defined by the coincidence of two ormore threads, and moving selectively 

w forward or backward in time through the displayed sequence of facts or events, and the method including ena- 
bling the selection, for subsequent display, of any one of the- threads of a displayed event 

The invention also provides an apparatus for recording facts relevant to the activities of the user of the 
apparatus, including a database, a user interface, and. means for receiving signals from external sources, the 
signals representing facts related to the sources; each of the facts being time-stamped and comprising an at- 

15 tribute and a value of that attribute for a given source at the time indicated by the time stamp, and the facts 
being automatically recorded in a chronological sequence in the database on the occurrence of physical or 
logical collocations of the apparatus and one or more of the sources; and the apparatus being operable through 
the user interface to enable the user to retrieve from the database a sequence of occurrences of a selected 
factor pattern of facts. 

20 Preferably, the apparatus of the preceding paragraph include means to enable manual entry of the facts 

or other information into the database, and is wearable by the user. 

The invention may be employed to provide a memory prosthesis irr the form of a highly portable device 

(e.g. a device worn like a watch) which records all kinds of information about the user's life, and provides a 

simple and rapid means of access to that information. The device receives and stores information from external 
25 sources ('objects') via, for example, a cellular communications network, or from signal generators identifying * 

specif ic machines, places, or people carrying other ones of the same type of device. The data can either be 

stored in the memory of the device itself, or, for better protection against failure, in a central data repository - 

of the system, with a fail-safe backup. 

Data is gathered continuously with no attempt being made to categorise or filter it at the time of collection. 
30 This means that all the data is retained, so that items that might have been categorised as insignificant at the ; . 

time of collection can later be retrieved. As a result there will be large quantities of data for the user to search. 

For the memory prosthesis to be usable, it must provide a mechanism for enabling the user to constrain the 

search and rapidly find the relevant information. This mechanism is supported by the data model described 

below. ■'-*.< • • 

35 The data consists of series of events which happen to each object in the system. Each object has its own 

"thread", a linear organisation in time of the events occurring as the object encounters other objects. Access 
to the data is achieved by displaying sequentially, running either forwards or backwards in time, the series of 
events along a selected thread. Rapid searching is made possible by specifying events formed by the coinck 
dence of several threads, or by changing from one thread to another at a event where the two threads intersect 
40 The memory prosthesis records ail kinds of information about the user's life, and by providing a simple 

and rapid means of access to this information it can help the user to recall things that might otherwise be for- 
gotten. All information in certain specified categories is gathered, without filtering, so that even things that 
might not have appeared important at the time of recording can later be recovered. 

A system and method in accordance with the invention will now be described, by way of example, with 
45 reference to the accompanying drawings, in which:- 

Fig. 1 is a diagrammatic representation of the elements which comprise an 'object", as defined herein; 
Fig. 2 is a pictorial representation of an example of a system in accordance with the invention; 
Fig. 3 is a diagrammatic representation of the threads of several objects, 

Fig 4 illustrates a set of 'threads' to show how a user can navigate the information in a database; 
so Fig 5 shows an example of a user interface; and 

Fig. 6 is a flow chart showing how data is processed by one of the 'objects' used in the system of the in- 
vention. 

Referring to Fig. 1 , the basis of the information recording system comprises at least two 'objects' 10 of the 
kind shown, which together define a d main. Th domain may cover, f r example, a room, a building, all the 
55 buildings within a group of buildings, or a larger area. An bj ct 10 represents a person or another identif iable 
entity such as a locati n, an apparatus or machine, a vehicl , a piece of mail, s m text, a numb r, a tim . 
The system thus comprises a set f obj cts 1 0, within th domain, for which a hist ry can b g nerated auto- 
matically. Each object 10 conforms to th archrtectur sh wn in Figur 1. All objects 10 hav a database 12 
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in which they sto^ 

database 12; it exchanges .nform^^^^^ 

all their information from remote sources over the ^™ ni ^ s f*™^ objects ^hfch form a domain 
Referring now to Ffcure 2, there is ^* s ^ an agent, a 

inciting a persona., portable, computer 30 ^^^^^^^ n ^ itonn ^ m ^ ar epre- 
database. a U. which includes a screen ^^n^om the^o represents, carries or wears the 

ferabty wireless, such as ^ 

tab as she moves around the doma.n «Mi outvano ^ g ^ 

(Jane's.Office). ^ " l ^ 

Bywayofexam P .e..et U sa^^^ 

will show -10:15 (time-stamp) - Jane (object at (a""™^ Ja ^ , f Steve enters Jane s 

on her tab can use either woras or icons » * S P^^ office. When Jane 

office at 10:47. a record is generated to show that Steve and Jane » «" va|ue ^ 
uses the telephone at 10:53 to call Bob. the ^^^^^^fki), in Jane's office. 
ephW. The record will show -^J-* database, either autc- 

^she^d^^^ 

her tab. Jane then finishes me.te.ephone^ 

If. many months later, and using the exarnp. calls. This mfcht be un- 
her telephone conversation, she can ask her tab*, * T ^ J epnone at « time when Steve 

DATA MODEL 

Information is held in the database as a sequence of time-stamped facts. 
Each fact consists of an object, an attribute and a value. 
Mac/represents the knowledge that the attribute of the object . the va.ue. 
The mlstamp indicates the time at which this fact Mm 

^T^r^ents the particular va.ue of an attribute, such as btock for hair colour, the kitchen for where- 
abouts, or larg for siz . (S Table 1 b low). 
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Time 


Object 


Attribute 


Value 


11:05:26 Mon 21st June 1993 


Steve 


at 


Kitchen 


11:37:13 Mon 21st June 1993 


Jane 


at 


Jane's Office 


1 1 :39:22 Mon 21 st June 1 993 


Jane 


at 


Kitchen 


11:42:35 Mon 21st June 1993 


Jane ■ 


at 


Jane's Office 


12:01:04 Mon 21st June 1993 


Jane 


availability ■ - , 


Busy 


12:01:09 Mon 21st June 1993 


Kitchen 


temperature 


22 Celcius 


Table 1, Typical time-stamped facts in a database. 



The episode which Table 1 represents can be described as follows. Steve enters the kitchen at 11:05. Jane 
enters her office at 11:37, and then goes into the kitchen at 11:39. She returns to her office" at 11:42. and is 
recorded as being 'busy* (e.g. with a telephone call) at 12:01 . Since ho change lias been noted in Steve's 'af at- 
tribute, it can be assumed that he remained in the kitchen throughout this period of time. It is possible to infer 
from this that while Jane was in the kitchen, Steve and Jane were collocated at 11 :39. This inferred fact is shown 
in Table 2. [ ' / " '\. ' ' / - ; _ ^ 
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INFERENCE 

An agent may apply inference rules to the information thus held, in order to establish other facts not directly 
recorded, such as when two people were in the same room, or the duration of a meeting. 

Such inferred facts may be determined by the agent upon request, or determined in advance and held in 
the database on the same footing as ordinary facts. 

Inferred facts may be used during inference, to produce further inferred facts. (See Table 2 below). 



Tirhe 


Object 


Attribute 


Value ■ 


1 1 :05:26 Mon 21 st June 1 993 


Steve 


at 


Kitchen 


1 1 :37: 1 3 Mon 21 st June 1 993 


Jane . , 


at . 


Jane's Office 


1 1 :39:22 Mon 21 st June 1 993 


Jane 


at ' 


Kitchen 


11:39:22 Mon 21st June 1993 


Steve 


co-located 


Jane 


11:42:35 Mon 21st June 1993 


Jane 


at 


Jane's Office 


12:01:04 Mon 21st June 1993 


Jane 


availability 


Busy 


12:01:09 Mon 21st June 1993 


Kitchen 


temperature 


22 Celcius 


Table 2. An extra inferred fact introduced, in italics 
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When a request for information is made to the agent a pattern may be specified. 

A pattern consists of either a conjunction of patterns; a disjunction of patterns; a negation of a pattern; 
or an object. 

Such a pattern filters the information yielded by the agent to include only those facts which conform to 
the pattern. 

A fact conforms to a conjunction of patterns if it conf rms to ach of the conjoined patterns individually. 
A fact conforms to a disjunction of patterns if it conforms to any one or more of th disjoined patterns. 
A fact conforms to a negated pattern if it does not conform to th pattern which is negated. 
A fact conforms to an object pattern, if th fact mentions the bject of th pattern. 
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SEARCH 

In iresoonse to information request, the agent searches for particular facts by c^nsrd^ 

pattl^^ possibly huge search undertaken by the.agent to reasonable 

dl To S reSmD»e a pattern which specified that a particular person, place and day of the week must be men- 

for conformity to the pattern. 
THREADS 

'^Simple user-interface paradigm wou.d be to keep at.eastone object in the pattern at alUimes. and dteplay 

biography displayed js now that ol -the obje cts. each one of which automatically generates 
The world is thus seen *" ^^jeS^id in principle be anything, but typicai exam- 

an autobiography record,ng events ,n rts life J^ ob ^ te , epn ones notebooks and workstations (see 

'msmsMmi 

to a u 0 ser e s can retrieve information about events in their life by finding the relevant points in the threa* ^ruc- 
and hence return the ong^als. ^ e numberof vents in any tnrea d 
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The actual notes, which may have been recorded in the database by the user, ar linked to these v nts, thus 

allowing him t retrieve them. 

By way of example of a user 'navigating' the database using the threads model, reference is made to Figure 

4 in conjunction with Table 3. Figure 4 shows a sequence of seven events, labelled in chronological order of 
5 their occurrence from 1 to 7. Each event comprises the conjunction (intersection) of two or more of threads A, 

B, and C. Event 1 shows a coincidence of threads A, B and C, event 2 involves only threads A and B, event 3 

involves threads A and C, arid so on. In searching the part of the database involving these events, the user 

might 'browse' in the manner represented by way of example in Table 3. In the first step, he starts at event 1 , 

moves forward (in time) along thread A and arrives at event 2. In step 2 he moves backwards. along thread A 
w and returns to event 1. In step 3 he changes to thread B. In step 4 he moves forward along thread B to arrive, 

on thread B, at event 2. In step 5 he moves forward along thread B to arrive at event 4. In step 6 he changes 

to thread C. In step 7 he moves forward along thread C to arrive at event 6. 

Instead of navigating along single threads, as just described, the user may select patterns of threads, the 

simplest pattern being a conjunction of two or more threads. I step 10, search is requested by the conjunction 
15 of threads A, B and C. Starting at event 1 , which is the first such conjunction, the user moves forward to event 

7, which is the next such conjunction. In step 11, a change is made to a pattern consisting of the conjunction 

of threads A and B. In step 12, the user moves backwards through such patterns,-the next occurrence of an 

A,B conjunction (going backwards) being event 5. 



Step No. 


Event 


Search Direction 


Thread ... 


Ftesult 


1 


1 


Forward 


A _ 


2 (on A) 


2 


2 


Backward 


- \ A - 


1 (on A) 


3 


1 


Change to B 


A 


1 (onB) 


4 


1 


Forward 


B 


2 (on B) 


5 


2 


Forward 


B 


4(onB) 


6 


4 


Change to C 


B 


4 (on C) 


7 


4 


Forward 


C 


6 (on C) 












.10 


1 


Forward 


A,B.C 


7 (on A, B, C) 


11 


7 


Change to A,B 


A, B, C 


7 (on A, B) 


12 


7 


. Backward 


A, B 


5 (on A,B) 


Table 3, Example of navigating data in a threads model 



USER INTERFACE 

Figure 5 gives one simple example of how the user interface rhight be organised for the memory prosthesis. 
The physical buttons on the side of the device can be used for moving forward or back through the current 
thread. Touching one of the on-screen buttons associated with intersecting threads switches to that thread. 

The device shown in Figure 5 f in the form of a 'tab', can be large enough for words to be used on its screen. 
However, if a smaller device, such as one in the same form as a watch, will need to have its information dis- 
played in a highly codified fashion. I this case, each episode or event listed occupies a single line containing 
a sequence of icons. The first item in each line represents the start time of the episode. It is followed by an 
activity indicator, representing the type of activity, such as a meeting, a time spent doing paperwork, or per- 
haps a vacation. The rest of the line contains a sequence of icons, each of which represents an obj ct which 
the user encountered in the displayed episode. Th top line of such a display contains information which applies 
to th whole of th history. It includes an icon representing th owner of the history displayed. It also includes 
th icons representing the filter pattern. The rest of the line contains the list f items that are common to all 
the episodes listed below it. This avoids repeating icons in each pisode description, thereby economising on 

7 
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screen space. ^ . rt ^ some other application, in a similar 

In reality any such user interface would be gen rally used as for th user to locate in- 
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and a user interface; „ n . or anc j at | ea st one user interface; 

ind ^acS b ^ 

locations of two or more of the objects. 

The^tem of daim 1 induding means to enab.e manua, entry of the facts or other information into the 

database. 

The system of daim 1 e, daim 2 f urther « 
at predetermined times. 

The system of any one of d.ims 1*3 further induding mea^ in the 

database on the occurrence of predeterminedvalues of an attnbute. 

A-thod of searching adata^^ 

a chronological sequence of time-stamped facts all of whK* J*^ 9 „ an xAM ^ 

of which a.so represent descriptors « M o^-^ f val ofThTa tLte for a given object, the 

quent display, of any one of the threads of a displayed event 

of that attribute for a given source at the time ™*™** b ***Xe^'<Ze of pnysica , or lo gical cd- 
ical.y receded in a chrondogica. sequence^ ^^^^^^.^^ 

50 fact or pattern of facts. 

7. The apparatus of claim 6 induding means to enabie manua. entry of the facts or other information into 
the database. 

8 Theapparatusofdaimeordaimrwhereintheuserinterfaceenabiesth user to search by dfedaying 

sequentially the occurrences of a selected fad or pattern. 
9. The apparatus of any on of daims 6 to 8 wher in each sequence of occurrences of a fact comprises 
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a 'thread', wherein each of the facts is represented by the coincidence of two or more of th threads, and 
wherein the user can search the database by selecting one thread, or by selecting the coincidence of a 
plurality of threads, or by moving from one thread to another. 

10. The apparatus of any one of claims 6 to 9 wherein the apparatus is wearable by the user. 

11. A system comprising the apparatus of any one of claims. 6 to 10 and a plurality of the sources arranged 
to cooperate with the apparatus. 

12. The system of claim 11 including a plurality of said apparatuses, at least some of the apparatuses com- 
prising at least some of said sources. 

13. The system of claim 11 or claim 12 wherein at least some of the sources comprise a cellular communica- 
tions network, enabling a device to determine its position in the network. 

14. The system of any one of claims 11 to 1 3 wherein the, or each, apparatus is in communication with a cen- 
tral data processing unit 

15. A system composing: * . ^ ! 

a plurality of signal sources; each signal source providing signals that identify the signal source; 
a first one of the signal sources being, portableso that the first signal source can-be located near a second 
one of the signal sources; the first signal source providing signals that indicate location of the first signal 
source. 

signal receiving circuitry for receiving signals from the signal sources and for providing signal data 
indicating information from the signals; upon receiving a signal from the first signal source indicating that 
the first signal source is located near the second signal source, the signal receiving circuitry providing 
collocation signal data indicating that the first signal source is located hear the second signal source; 

memory for storing data; and 
processing circuitry connected for receiving signal data from the signal receiving circuitry and also con- 
nected for accessing the memory; 

upon receiving signal data indicating a signal from one of the signal sources, the processing cir- 
cuitry using the signal data to obtain event data, the event data indicating the signal source and a time 
at which the signal data were received; 

upon receiving the collocation signal data, the processing circuitry using the collocation signal data 
to obtain collocation event data, the collocation event data. indicating the first signal source, the second 
signal source, and the time at which the collocation signal data were received. 
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(54) Recording and retrieval of information relevant to the activities of a user. 



\ A method for the recording and retrieval of 
information relevant to the activities of the user. 
The invention may be employed to provide a 
memory prosthesis in the form of a highly port- 
able device (30) (e.g. a device worn like a watch) 
which records all kinds of information about the 
user's life, and provides a simple and rapid 
means of access to that information. The device 
receives and stores information from external 
sources ('objects') via, for example, a cellular 
communications network, or from signal 
generators., identifying specific machines, 
places, or people carrying other ones of the 
same type of device. Events comprising en- 
counters between 'objects' are recorded auto- 
matically, being time-stamped and stored 
chronologically, and a rapid search method for 
such a database is disclosed. The data can 
either be stored in the memory of the device 
itself, or, for better protection against failure, in 
a central data repository of the system, with a 
fail-safe backup. 
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